In addition to NMF, we'll also be implementing a more novel approach, known as hierarchical Poisson factorization (HPF) \cite{gopalan2015scalable}. HPF was developed as an improvement to the recently popularized (by the Netflix Prize) recommendation problem. The algorithm has its roots in NMF, but makes some significant modifications. In the recommendation setting of the problem, HPF takes in a user behavior matrix, which is a matrix of users as rows, and items that have been consumed as columns. For recommendation, we have a binary matrix (1 = consumed, 0 = not consumed), however, we can use continuous ratings instead.
The data are modeled as factorized Poisson distributions, where each user has K latent preferences and each item has K latent attributes. The observations (entries in matrix) are modeled as a Poisson distribution, parameterized by the inner product of the corresponding user and item latent value vectors. Given this matrix, the basic generative model is as follows:
- For each user u:
- For each latent variable k:
- sample preference $\theta_{uk} \sim Gamma$
- For each latent variable k:
- For each item i:
- For each latent variable k:
- sample attribute $\beta_{ik} \sim Gamma$
- For each latent variable k:
- For each u and i, sample rating/consumption:
- $y_{ui} \sim Poisson$
Each of the Gamma distributions above are also further parameterized by Gamma distributions. For simplicity, we've removed the parameters above, but the plate notation diagram is as below:
There are a number of similarities between our text corpus and the user behavior matrix
| HPF Attribute | User Behavior | Corpus Topic Modeling |
|---|---|---|
| Additional Priors on latent variable Gammas | Captures the diversity of users, some tending to consume more than others, and the diversity of items, some being more popular than others | Some documents contain many topics while some are about a singular topic. Some topics are more prevalence than others. |
| Estimation of Posterior Expectation | Each user’s preferences and each items attributes | Each document's topics and each word's semantics |
| Model of Long-Tailed Behavior | While most users consume a handful few items, a few “tail users” consume thousands of items | While most documents only consist of a few topics, some documents may be review articles that have to do with many topics |
| Downweights effects of 0s | A user that consumes an item must be interested in it so 1s are more relevant than 0s | There are lots of words that are completely unrelated to a document, while a few are very relevant (within the corresponding topic). We should focus on these occurrences rather than the lack of most words |
In the last example, normal randomized MF uses Gaussian likelihoods which weight seen and unseen examples equally, which in a sparse matrix, means most factors relie on the "unconsumed" user/item pairs. HPF corrects for this problem.
